Sadhika Malladi
I am a PhD student in computer science at Princeton University, where I work on theoretical machine learning with Sanjeev Arora. My work focuses on using mathematical insights into deep learning (especially language models) to design and analyze performant and efficient algorithms.
I graduated from MIT in 2019 with a degree in mathematics with computer science (18C) and a degree in philosophy (24-1). During my undergraduate studies, I also worked on language modeling at OpenAI in 2018 and 2019.
You can reach me at [my first initial][my last name] (at) princeton (dot) edu.
In my free time, I like to ride my bike, read, and watch sports. My Google scholar page is here. I adapted this website from Gregory Gunderson.
Selected Works
My work aims to advance ML theory, especially in optimization, to understand LLMs and design better algorithms to train them.Fine-Tuning Language Models with Just Forward Passes
Sadhika Malladi*, Tianyu Gao*, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, Sanjeev Arora (Oral at NeurIPS 2023).
We propose MeZO, a theoretically motivated memory-efficient zeroth-order optimizer to fine-tune LLMs, and we demonstrate that it can often effectively fine-tune models using up to 12x memory and half as many GPU-hours.
LESS: Selecting Influential Data for Targeted Instruction Tuning
Mengzhou Xia*, Sadhika Malladi*, Suchin Gururangan, Sanjeev Arora, Danqi Chen (ICML 2024).
We develop a model- and optimizer-aware data selection algorithm that permits using just 5% of the data to outperform using the entire dataset during instruction tuning.
Preference Learning Algorithms Do Not Learn Preference Rankings
Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, Kyunghyun Cho.
We demonstrate that preference learning algorithms rarely cause models to place higher likelihood on the preferred response over the dispreferred response.
A Kernel-Based View of Language Model Fine-Tuning
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, Sanjeev Arora (ICLR 2021).
We formally describe the optimization dynamics of language model fine-tuning as kernel behavior and verify this hypothesis through extensive experiments. Our theory provides insight into why parameter-efficient methods like LoRA work, and it motivated the development of MeZO and LESS.
SDEs and scaling rules for SGD, SGD with momentum, RMSprop, and Adam.
Different author lists, published at NeurIPS 2021, NeurIPS 2022, and ICLR 2024.
Our work with stochastic differential equations (SDEs) describes the optimization dynamics of training (nearly) arbitrary deep models from (nearly) arbitrary initialization on (nearly) arbitrary data using SGD, SGD with Momentum, RMSProp, and Adam. The theory lets us set hyperparameters to enable efficient learning in highly distributed settings (i.e., large batch size). My blog post provides an accessible introduction to SDEs and the resulting practical implications for training vision and language models.